Method for simultaneous capture of image data at multiple depths of a sample

ABSTRACT

A novel method is disclosed to allow for the simultaneous capture of image data from multiple depths of a volumetric sample. The method allows for the seamless acquisition of a 2D or 3D image, while changing on the fly the acquisition depth in the sample. This method can also be used for auto focusing. Additionally this method of capturing image data from the sample allows for optimal efficiency in terms of speed, and light sensitivity, especially for the herein mentioned purpose of 2D or 3D imaging of samples when using a tilted configuration as depicted in FIG.  2 . The method may be particularly used with an imaging sensor comprising a 2D array of pixels in an orthogonal XY coordinate system where gaps for electronic circuitry are present. Also other imaging sensor may be used. Further, an imaging device is presented which automatically carries out the method.

CROSS-REFERENCE TO PRIOR APPLICATIONS

This application is a Continuation application of U.S. application Ser.No. 15/124,610, filed Sep. 8, 2016, which is the U.S. National Phaseapplication under 35 U.S.C. § 371 of International Application No.PCT/EP2015/079331, filed on Dec. 11, 2015, which claims the benefit ofEuropean Patent Application No. 14199531.6, filed on Dec. 22, 2014.These applications are hereby incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to the field of imaging a sample, andapplies advantageously in the field of digital pathology.

In particular, the present invention relates to a method forsimultaneously capturing image data at multiple depths of a sample andto an imaging system for simultaneous capture of image data of a sampleat multiple depths.

BACKGROUND OF THE INVENTION

A digital scanning microscope usually makes a digital image of a samplesuch as a tissue sample placed in a microscope slide. This is typicallydone by scanning the sample over the whole microscope slide andstitching different image spans together and/or by overlaying imagesmeasured at different wavelengths. FIG. 1 schematically represents across-section 100 of such a microscope slide. A glass slide 101, a coverslip 102 and a mounting medium 103 for fixing and sealing off a sample104, like e.g. a biological tissue layer, are comprised. It is known,for example, from WO 2001/084209, that digital scanning microscopes cancomprise a 2D line sensor, also known as a line scan camera or as alinear array sensor. Such sensors comprise only one line, saiddifferently one row, of sensing pixels. It is also known that comparedto other types of sensors, like 2D array sensors, for example, 1D linesensors are able to provide better continuous mechanical scanningoperation, less stitching problems, and can allow for the use ofso-called time delay integration (TDI) line sensors.

Furthermore, current imaging sensor designs provide photoactive pixelswhich consist of photosensitive parts, i.e. photodiodes, and alsocomprise non-photosensitive parts like a plurality of charge to voltageconverters (CVC) that are embedded in the pixel itself leading to alower fill factor. This means that the pixel typically has threetransistors (rolling shutter) of four transistors (global shutter) forthe CVC and both vertical and horizontal metal lines for addressing andread out are needed. However, such non-photosensitive parts of the pixelreduce the fill factor of the pixel which is especially harming duringlow light conditions. The resulting low light sensitivity of pixels in aconventional sensor is typically overcome by applying microlenses. Suchmicrolenses try to effectively focus less amount of light onto thepixels of the imaging sensor such that the collateral loses areminimized. In addition, currently available imaging sensors provide fora relative low speed in reading out the region of interest (ROI) as onlya limited number of read out electronics can be provided within thelimited space of a given pixel size.

SUMMARY OF THE INVENTION

The inventors of the present invention have realized that the use ofmicrolenses is particularly not suitable when the imaging sensor istilted with respect to the optical path, which is applied often, interalia, in digital pathology. Moreover, the inventors of the presentinvention have found that scanning and imaging the sample with a tiltedsensor leads to an oversampling in Z direction such that only particularareas of the imaging sensor need to be used for image capturing. Hence,the inventors of the present invention found that one may use duringimaging only pixel lines of the 2D imaging device or imaging sensorwhich are offset with respect to each other along the scan direction byan offset. This offset may, as example, either be a non-photosensitivegap as depicted in FIGS. 3 to 5, or may be one or more deactivated linesof pixels which are currently not used for image generation. For such animaging device a novel imaging method is presented herein and detailsthereabout will be explained in the context of several differentexemplary embodiments. The object of the present invention may be seenas providing for an improved method and system for image capturing.

The object of the present invention is solved by the subject-matter ofthe independent claims. Further embodiments and advantages of theinvention are incorporated in the dependent claims.

The described embodiments similarly pertain to the method for imagecapturing and the imaging system.

According to an exemplary embodiment of the present invention a methodfor simultaneous capture of image data at multiple depths of a sample ispresented. The method uses an imaging device having an optical axis, andthe imaging device comprises an imaging sensor tilted with respect tothe optical axis. The imaging sensor that is used in the presentedmethod has a first pixel line comprising a plurality of pixels and asecond pixel line comprising a plurality of pixels. The first and secondpixel lines have a different optical path length to the sample along theoptical axis of the imaging device, and the first pixel line and thesecond pixel line are offset with respect to each other along the scandirection by an offset. The method comprises the steps of scanning thesample along the scan direction (X′) which is substantiallyperpendicular to the optical axis and perpendicular to a main extensiondirection (Y) of the first and second pixel lines, capturing a firstimage of the sample from the first pixel line, and simultaneouslycapturing a second image of the sample from the first pixel line andcapturing a third image of the sample from the second pixel line.Moreover, continuing capturing images of the sample from the secondpixel line is a further step of the method as well as stopping capturingimages of the sample from the first pixel line.

Therefore, a read-out method for generating seamless 2D or 3D imageswhile changing capture depth during the scanning is presented. Thisallows a fast image acquisition of not perfectly flat and/or volumetricsamples and can be specifically applied in digital slide scanners, e.g.for digital pathology, but also in other technical fields. With thismethod it is possible to capture a seamless image, which would not bepossible without the temporary dual, i.e., simultaneous readout, becausea change in line sensor results not only in a change of acquisitiondepth, but also in a translation along the scan direction. This lattertranslation causes either a gap in the image, or a repetition of imagedata. The method as described before is necessary to prevent the gap.For the repetition, part of the image data can be discarded, but no dualacquisition is needed. It is important to stress, that this method wouldalso be required when changing ROI in a conventional 2D CMOS sensor whenit's used for the purpose of a 2D autofocus system. This is because anychange in the ROI that leads to an effective change in acquisition depthwill also lead to a translation along the scan direction, if the 2D CMOSsensor is tilted with respect to the optical axis. This translationalong the scan direction will need to be compensated, if an undistortedfinal image is to be obtained. The present invention avoids suchdistortions as explained herein. The presented method can beautomatically carried out by an imaging device as disclosed herein.

Of course more lines of pixels than the first and second line can beused by this method for capturing image data. As can easily be gatheredfrom the embodiment examples shown in FIGS. 3 to 5, a large plurality ofpixel lines/line sensors, which are all respectively offset from eachother, can be used.

As is apparent for the skilled reader from this disclosure, each pixelof a pixel line captures an image, and subsequent processing generatesan image that is captured by the pixel line.

In general, two cameras, i.e., at least the two lines of pixels, areprovided at different distances from the sample such that they focus ata different depth in the sample.

Between these two cameras the “offset” as defined below is located. Alsoa large 2D sensor can be used as will be explained in more detailhereinafter and as is show in e.g. FIG. 3.

The term “offset” or “gap” as used in the context of the presentinvention shall be understood as a space or distance between twoneighbored lines of pixels which space is not photoactive. This spacemay, for example, be used for placing read-out electronics in such areasof the sensor or may be embodied by one or more lines of pixels whichare currently not activated and thus not photoactive as the pixels inthe gap are just not used. The offset is currently not capturing animage.

Thus, a novel method is disclosed to allow for the simultaneous captureof image data from multiple depths of a volumetric sample. The methodallows for the seamless acquisition of a 2D or 3D image, while changingon the fly the acquisition depth in the sample. This method can also beused for auto focusing. Additionally this method of capturing image datafrom the sample allows for optimal efficiency in terms of speed, andlight sensitivity, especially for the herein mentioned purpose of 2D or3D imaging of samples when using a tilted configuration as depicted inFIG. 2.

For example, this method can be applied by an imaging sensor of animaging device which combines multiple TDI line sensors on a single die,which will be explained in detail in the context of the embodimentsshown in FIGS. 3 to 5. There, the TDI line sensors are separated by agap. Such a sensor may have a dual (TDI) read-out engine, which allowsefficient reading out of at least two line sensors at maximum speed andsensitivity. The improvement over a conventional 2D sensor of the samesize and resolution achieved by this novel design and read out method istwofold. First, the gap between the photo sensitive parts/lines ofpixels (the TDI linesensors) can be used to put the logic and connectivecircuitry of the sensor. This allows for maximizing the photo activepart of the pixels in the photo sensitive area of the sensor, i.e.maximize the fill factor. This allows for a sensitive sensor withoutmicrolenses that are common on 2D CMOS sensors. Avoiding micro-lenses isimportant for placing the sensor tilted in the optical path. Second, thegap allows for faster read-out because more circuitry can be on thesensor in the gap, allowing for a faster read out method and a fastersensor.

As will be easily understood by a person skilled in the art, theinvention may be not be limited to a configuration of the system inwhich the sensor should be tilted with respect to the optical axis. Theinvention obviously encompasses other configurations in which the sensoris not tilted and in which the imaging system is arranged such that thissensor can image an oblique cross section of the sample. Thus, thegeneration of said different optical path lengths from the sample to thesensor may be made using other techniques well-known in the art such asinserting an optical element, for instance a prism, in the light path.

The method of simultaneous capture of image data at multiple depths of asample overcomes two problems that result from normal imaging methodsusing a normal 2D CMOS sensor for the autofocus and 3D imaging. On theone hand low light sensitivity due to low fill factor of pixels in anormal 2D CMOS sensor can be improved. This is normally overcome withmicro-lenses, but micro-lenses are not suitable for use when the sensoris tilted with respect to the optical path as shown in FIG. 2. Moreover,the low speed in ROI read-out of normal 2D CMOS sensors can be increasedby the present invention as more read-out electronics can placed in thespaces between first pixel line and the second pixel line, as describedin more detail, for example, in the context of FIGS. 4 and 5. Accordingto another exemplary embodiment of the present invention the offset iseither a first non-photosensitive gap between the first and the secondpixel lines or is one or more non-capturing pixel line(s) between thefirst and the second pixel lines which non-capturing pixel line(s) is(are) deactivated.

According to another exemplary embodiment of the present invention thenon-photosensitive gap extends parallel to the first and second pixelline. As will be explained in the context of FIG. 3, this direction willbe named Y direction. The method of the present invention can be usedwith an imaging sensor that comprises a 2D array of pixels in anorthogonal XY coordinate system, the 2D array of pixels of the sensorcomprising a plurality of pixels, and each of the pixel lines extendsalong the Y direction.

As will become apparent from and elucidate with the exemplaryembodiments depicted and explained in the context of FIGS. 3 to 5 thesteps of the method for simultaneous capture of image data at multipledepths of the sample are carried out during scanning the sample by ascanning imaging system. Such a scanning imaging system is anotherexemplary embodiment of the present invention.

According to another exemplary embodiment of the present invention thesimultaneous capturing of the second image and the third image iscarried out for as long as it takes to bridge the offset between thefirst pixel line and the second pixel line during scanning. Therefore,the read-out method generates seamless 2D or 3D images while changingcapture depth during the scanning. Clearly, this allows a fast imageacquisition of not perfectly flat and/or volumetric samples. Thecalculation to determine how long it takes to bridge the respectiveoffset can be done by the skilled person without a problem. Startingfrom the distance between the two cameras, i.e., the two pixel lines, inprojection one knows to how many pixels the distance relates. With theexposure frequency, i.e., the line rate, one knows how many pixels arein the offset/gap. In another exemplary embodiment it is also possibleto do real time detection if for example the sample position mightfluctuate during scanning. One example that might require real timedetection would be the case where there is no scanning, but a flow in afluid. This flow might be less regular, which means that it's not afixed amount of exposures before the gap is bridged. The real timedetection in this case would be tracking of the object's lateralposition as it flows by.

According to another exemplary embodiment of the present invention themethod further comprises the steps of detecting whether a change inacquisition depth is required, and actuating the second pixel line basedon the detection that a change in acquisition depth is required. Manydifferent technical means can be used for the detection whether a changein acquisition depth is needed. For example, focus signal detection asknown in the art of imaging, detection in an additional optical path,e.g. a confocal microscope, previous knowledge of shape and/ororientation of the sample, or methods of predicting the optimal focusposition can be used to determine that and which new line or lines needto be activated for image capturing. As such methods are already knownto the skilled person they are not described in greater detail herein.

According to another exemplary embodiment of the present invention theimaging sensor which is used for the method presented herein furthercomprises a third pixel line comprising a plurality of pixels, whereinthe first, second and third pixel lines each have a different opticalpath length to the sample along the optical axis of the imaging device.Further, the first pixel line and the third pixel line are offset withrespect to each other along the scan direction (X′) by an offset and thefirst pixel line is located between the second and the third pixel line.With this imaging sensor and according to the method the second pixelline is activated in case it has been detected that an increase inacquisition depth is required whereas the third pixel line is activatedin case it has been detected that a decrease in acquisition depth isrequired.

In other words, first an image from the first line sensor is captured,second it is detected that a change in acquisition depth, i.e., changeacquisition from current line sensor to the one above or below is neededand third capturing two images simultaneously from the current and thenew, either above or below, line sensor, for as long as it takes tobridge the gap between the two line sensors at the current scan speed.Subsequently, it is continued capturing images or image data from thenew line sensor, and capturing from the initial line sensor is stopped.With this flow it is possible to capture a seamless image.

According to another exemplary embodiment of the present inventiondiscarding repetition data acquired by the first pixel line and/or thesecond pixel line is part of the method. In case an overlap of imagedata has been captured during the step of simultaneously capturing imagedata one part may be discarded or deleted. The same calculation as givenbefore with respect to the determination of the time how long it takesto bridge the offset or gap between two pixel lines can be applied here.One may either throw away the data that was already captured, wait untilthe activated line is at a zone during scanning which has not beenimaged, or one may combine these two alternatives.

According to another exemplary embodiment of the present inventiongenerating a final image of the sample based on the captured imagesafter the repetition data was discarded is part of the method.

According to another exemplary embodiment of the present invention amethod as described before is presented which is a method for generatinga three dimensional (3D) image of the sample. This 3D imaging methodcomprises the steps of capturing the first image of the sample from afirst set of lines of pixels comprising the first pixel line, andsimultaneously capturing the second image of the sample from the firstset of lines of pixels and capturing a third image of the sample from asecond set of lines of pixels comprising the second pixel line.Moreover, the steps of continuing capturing images of the sample fromthe second set of lines of pixels, and stopping capturing images of thesample from the first set of lines of pixels are comprised.

According to another exemplary embodiment of the present invention animaging system with a first and second pixel line each comprising aplurality of pixels is presented. The device is configured to scan thesample along a scan direction (X′) and the first pixel line and thesecond pixel line are offset with respect to each other along the scandirection by an offset. Further, the imaging system is configured tocapture a first image of the sample from the first pixel line, and isconfigured to simultaneously capture a second image of the sample fromthe first pixel line and capturing a third image of the sample from thesecond pixel line. Moreover, the imaging system is configured tocontinue capturing images of the sample from the second pixel line andis configured for stopping capturing images of the sample from the firstlines of pixels. Embodiments thereof will be explained in more detail inthe context of the following Figures.

The imaging system has an imaging sensor comprising the mentioned pixellines, wherein the imaging sensor is titled with respect to the opticalaxis of the imaging system.

According to another exemplary embodiment of the present invention thefirst pixel line of the imaging system is part of a first block thatconsists of several adjacent pixel lines extending along the Ydirection, and the second pixel line is part of a second block thatconsists of several adjacent pixel lines extending along the Ydirection. Further, the first and second blocks are separated from eachother by a non-photosensitive gap extending along the Y direction. Sucha TDI embodiment can be gathered from FIGS. 4 and 5.

According to another exemplary embodiment of the present invention theimaging system does not comprise microlenses.

According to another exemplary embodiment of the present invention eachoffset or non-photosensitive gap has a width of at least one width of apixel of the used imaging sensor.

According to another exemplary embodiment of the present invention ascanning imaging system is presented, wherein the system is a digitalscanning microscope for imaging a sample.

According to another exemplary embodiment of the present invention inthe scanning imaging system the imaging sensor is tilted around the Yaxis as an axis of rotation. According to another exemplary embodimentthe method as presented herein is used in/by a digital scanningmicroscope to generate an image of a pathology sample.

These and other features of the invention will become apparent from andelucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the invention will be described in thefollowing drawings.

FIG. 1 schematically shows a cross-section of a microscope slide.

FIG. 2 schematically shows a scanning microscope according to anexemplary embodiment of the present invention.

FIG. 3 shows a projection of an imaging sensor in object space accordingto an exemplary embodiment of the present invention.

FIG. 4 schematically shows an imaging sensor using TDI principles andthe method according to an exemplary embodiment of the presentinvention.

FIG. 5 schematically shows a setup with an imaging sensor using themethod according to an exemplary embodiment of the present invention.

FIG. 6 schematically shows a flow diagram of a method according to anexemplary embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

According to an exemplary embodiment of an imaging system of the presentinvention a scanning microscope 200 is shown within FIG. 2. The imagingsystem 200 can carry out the method for simultaneous capture of imagedata at multiple depths of the sample as described herein. Particularly,the imaging system 200 is configured for carrying out the steps S1 to S5as disclosed in the context of FIG. 6. However, it is important to notethat the imaging system 200 allows for a read-out method for generatingseamless 2D or 3D images while changing capture depth during thescanning. This allows a fast image acquisition of not perfectly flatand/or volumetric samples. With this method and imaging system 200 it ispossible to capture a seamless image, which would not be possiblewithout the temporary dual, i.e., simultaneous readout, because a changein line sensor results not only in a change of acquisition depth, butalso in a translation along the scan direction. This latter translationcauses either a gap in the image, or a repetition of image data. Themethod of the present invention prevents the gap. For the repetition,part of the image data can be discarded, but no dual acquisition isneeded.

Of course the scanning imaging system 200 is arranged for imaging asample, e.g. a tissue layer which is not shown in FIG. 2, which can beplaced between the glass side 201 and the cover slit 202. Imaging path Pmay comprise a microscope objective 206 which may comprise one or morelenses 203, 204 and 205, an aperture 207 for blocking unscatteredreflected light from the tissue sample, a tube lens 208 and an imagingsensor 209 according to the present invention. Imaging sensor 209comprises a 2D array of pixels that can also be referred herein as amatrix of pixels. For example, the sensor is a CMOS imaging sensor butalso other kinds of sensors can be used with the present invention. Ascan be seen from FIG. 2, imaging sensor 209 is tilted with respect tothe optical axis o of the microscope objective lens. The imaging sensor209 may be a self-focusing imaging sensor as explained herein. Thesystem 200 further comprises a control module for controlling theoperating process of the scanner, and in particular the scanning processfor imaging the sample. The control module typically comprises aprocessor such as, for example, an FPGA (Field Programmable Gate Array)or a DCP (Digital Signal Processor). It should be noted, that theoptical axis O can be parallel to the axis Z 309 that is defined in thefollowing FIG. 3.

The method of the present invention can for example be carried out withan imaging sensor 300 as shown in FIG. 3, and which will be explained inmore detail hereinafter. In an exemplary embodiment the method comprisesthe steps of capturing an image from one line sensor/pixel line 304,detecting by whatever means that a change in acquisition depth, i.e.,change acquisition from current line sensor/pixel line 304 to the oneabove or below 310 is needed, and capturing two images simultaneouslyfrom the current 304 and the new line 310, for as long as it takes tobridge the non-photosensitive gap 305 c between the two line sensors atthe current scan speed. As a further step continuing capturing from thenew line sensor 310, and stop capturing from the initial line sensor 304is comprised. With this method it is possible to capture a seamlessimage, which would not be possible without the temporary dual readout,because a change in line sensor results not only in a change ofacquisition depth, but also in a translation along the scan direction.This latter translation causes either a gap in the image, or arepetition. Advantageously this method prevents the gap in the finalimage. For the repetition, part of the image data can be discarded, butno dual acquisition is needed. Detailed aspects about discarding imagedata have been described before.

Regarding the sensor used for this method FIG. 3 shows a projection 300of an imaging sensor 311. Moreover, imaging sensor 311 may be aself-focusing imaging sensor. FIG. 3 shows that the imaging sensor 311comprises several TDI blocks 304, 310 that respectively comprise aplurality of parallel pixel lines running along the Y direction shownwith axis 308. The TDI blocks 304 and 310 are separated by anon-photosensitive gap 305 c, in which the read out electronics of thepixels of at least one of said blocks are positioned. If desired, theread out electronics of both TDI blocks 304 and 310 can be positioned inthe gap 305 c. However, it is also possible, that the read outelectronics of the pixels of block 310 are positioned in thenon-photosensitive gap 305 c and that the read out electronics of thepixels of block 304 are positioned in non-photosensitive gap 305 a.Apparently it is possible to provide TDI blocks 304 and 310 as a pixelline which does not comprise charge to voltage converters and/or logicsand/or connective circuitries. The latter components are entirelycomprised by said non-photosensitive gaps of the imaging sensor 311 suchthat a maximization of the fill factor is achieved with a proper lowlight sensitivity is achieved. It should be noted, that the TDI blocks304 and 310 are only illustrated schematically such that the pluralityof adjacent pixel lines is not depicted here in detail. Such individualpixel lines constituting the TDI block may be gathered from followingFIG. 4. Also the gaps 305 a, 305 b, and 305 c are only schematicallydrawn within FIG. 3 301 denotes a glass slide and 302 denotes a coverslip and the tissue sample is shown with 303. Moreover, the scandirection X′ is depicted with arrow 306 and it can easily be gatheredthat the scan direction X′ is substantially perpendicular to the Ydirection 308 defining the 2D array of pixels of sensor 311. X direction307 is also shown in FIG. 3.

The imaging sensor of FIG. 3 has a two-fold improvement over aconventional 2D sensor of the same size and resolution. Maximization ofthe photoactive part of the pixels in the photosensitive area of thesensor is provided such that the fill factor is maximized. This allowsfor a sensitive sensor without microlenses. Avoiding microlenses isimportant for placing the sensor tilted in the optical path of, forexample, a scanning imaging microscope. Furthermore the gaps 305 a, 305b, and 305 c allow for a faster read out because more circuitry can beon the sensor in the gaps allowing for a faster sensor.

FIG. 4 schematically shows an imaging sensor 400 according to anexemplary embodiment of the present invention. The imaging sensorcomprises a 2D array of pixels 421. The 2D array of pixels comprises afirst pixel line 410 which comprises a plurality of pixels, for examplepixels 415, 416, 417. As can be gathered from FIG. 4, the first pixelline 410 extends along the Y direction 422 from the left end of thearray to right end if the array, thus it extend over the whole breadthof the array. The Y direction is perpendicular to the X direction 423.The 2D array of pixels further comprises a second pixel line 411comprising a plurality of pixels, pixels 418 and 419 are exemplarilyshown with reference signs. Also the second line 411 extends also alongthe Y direction 422. Furthermore, a first non-photosensitive gap 402between the first and the second pixel lines is provided. As can begathered from FIG. 4, this gap also extends along the Y direction.Moreover, read out electronics 412 and 413 of pixels of the first lineand/or of pixels of the second pixel line are positioned within thefirst non-photosensitive gap 402. Of course components, like e.g. anFPGA, may be comprised by the sensor.

The first non-photosensitive gap 402 has a width of at least one width420 of a pixel of the sensor. In this embodiment, the gap width isapproximately five times the width 420 of one pixel of the sensor. Alsothe second non-photosensitive gap 403 has such a width. Moreover, as canbe gathered from FIG. 4, a first block 405 of several adjacent pixellines 407 to 410 are comprised by the imaging sensor 400. This firstblock 405 can be controlled as a TDI block. The first non-photosensitivegap 402 comprises read out electronics 412, 413 such as current voltageconverters of the pixel line 410 or of line 411 and may also comprise alogic of the imaging sensor 400 and/or a connective circuitry of theimaging sensor 400. Also the second non-photosensitive gap 403 comprisessuch read out electronics 414 of pixels of the second TDI block 404 andof third TDI block 406. As has been described before, the imaging sensor400 can also be provided such that the read out electronics of a TDIblock are completely provided within the adjacent gap below or abovesaid TDI block along the shown X direction 423. It is thus possible toprovide for a TDI block that consists entirely of photodiodes but doesitself not comprise read out electronics as they are moved to neighboredgaps. In another exemplary embodiment the sensor has 128 of such blocks404, 405, and 406 of pixel lines and has 127 or 128 gaps.

A TDI block may be seen as 2D array of pixels with the lines/rowsextending along the Y direction, and the columns extending along the Xdirection. The TDI action takes place along the columns. This TDI actioncan either be a conventional CCD fashion TDI, where the charge istransferred along the columns synchronized with the motion of the objectwith respect to the sensor. Alternatively, TDI in the digital domain canbe carried out, where the pixel charges are first converted to a digitalnumber, and then transferred in the digital domain synchronized with themotion of the object with respect to the sensor. This ‘digital’ TDI cantake place on the image sensor itself, or ‘off-chip’, in a computationalunit such as an FPGA or computer. The system of the present inventionmay also comprise a control module which controls the read out of theimaging sensor such that the desired TDI procedure takes place.

A more detailed example of using TDI according to such embodiments isexplained in the context of FIG. 4. In FIG. 4, three blocks 404, 405 and406 of four TDI stages (e.g. 407, 408, 409, 410) are designated in thepixel matrix. Note that a TDI block is meant to be a sub-array of thetotal pixel matrix, which acts as a functional TDI unit. A personskilled in the art will derive in an obvious manner how a TDI sensoraccording to such embodiments may operate. Some embodiments will bedescribed herein by way of non limitative examples. All of them areapplicable to both of the two dominant imaging sensor types, i.e. CCDand CMOS image sensors. For CCD image sensors the TDI action istypically executed in the analog domain, by copying charge from one setof pixels to another set of pixels. For CMOS image sensors, the TDIaction is typically performed in the digital domain, by adding thedigital value of one set of pixels to the digital value of another setof pixels. However, digital and analog TDI can both be applied to eitherof CCD and CMOS.

In the following the TDI action is described as a pixel value transfer,which is to be understood as an analog charge transfer if analog TDI isemployed, and as a pixel value transfer if digital TDI is employed.

Turning back to the example of FIG. 4, the sensor is moved to a scanposition further with respect to the microscope slide while a pixelvalue is transferred. In the example of FIG. 4 it will be assumed thatthe TDI action works upward and the translation of the sample withrespect to the sensor is made upward too. Pixel line or stage 410 (astage preferably includes a full pixel line) starts with pixel values of0 for each exposure, and pixel values from stage 407 make up the finalimage in block 405 after each exposure. When following a single line ofthe image of the sample during a full TDI cycle, the process, which isknown in the art, is as follows: during an exposure at a time t=0, animage of the sample is captured by the imaging sensor. At the nextexposure at t=1, the sample is translated such that the part of theimage of the sample projected at t=0 on stage 410 is now projected onstage 409. Between exposures t=0 and t=1, the values of the pixels instage 410 are copied to stage 409. During the exposure at t=1, the pixelvalues resulting from the exposure on stage 409 are added to the alreadypresent values, which resulted from the exposure at stage 410 at t=0.The values in stage 409, are now the sum of the pixel values resultingfrom the exposure of stage 410 at t=0 and the exposure of stage 409 att=1. Between exposures t=1 and t=2, the values of the pixels in stage409 are copied to stage 408. During the exposure at t=2, the pixelvalues resulting from the exposure on stage 408 are added to the alreadypresent values, which resulted from the exposure at stage 410 at t=0plus the exposure at stage 409 at t=1. The values in stage 408, are nowthe sum of the pixel values resulting from the exposure of stage 410 att=0 and the exposure of stage 409 at t=1, and the exposure of stage 408at t=2. Between exposures t=2 and t=3, the values of the pixels in stage408 are copied to stage 407. During the exposure at t=3, the pixelvalues resulting from the exposure on stage 407 are added to the alreadypresent values, which resulted from the exposure at stage 410 at t=0plus the exposure at stage 409 at t=1, and stage 408 at t=2. The valuesin stage 407, are now the sum of the pixel values resulting from theexposure of stage 410 at t=0 and the exposure of stage 409 at t=1, andthe exposure of stage 408 at t=2, and the exposure of stage 407 at t=3.Because the image of the sample is translated over the sensor in thesame direction, and at the same speed as the TDI action, in this examplefour equal exposures have been made of the same area on the sample. Thisis equivalent to a four times longer exposure period without slowingdown the translation of the sample and without introducing additionalmotion blur. The above description applies as well to any other blockssuch as blocks 404 and 406 or any further block of the imaging sensor ofthe present invention.

It is to be noted that in such embodiments the four stages of the TDIblocks may be able to capture an image of the same area at same focus.

Accordingly, the stages of each TDI block may be such that they areseparated from the sample by the same distance, approximately.

For example by referring back to the first detailed implementationdescribed above, four stages can be used for each block. Thus, each ofthe TDI blocks may be constituted by four lines of pixels positionednext to each other with a pitch having the same size as the pixel size.It is to be noted here that a pitch may refer to the distance betweenthe centers of two neighboring pixels. Each TDI block in each embodimentof the present invention may be spaced apart by a non-photosensitive gapdistance larger than the pitch. The gap distance determines the Zresolution of the depth positioning of the sensor. It may beadvantageous to have a relatively large gap, while having the individualpixels of each TDI block closer together. In this manner a relativelylarge Z range can be obtained without using too many pixels, because theindividual stages of each TDI stage are closer together. As a resultthey acquire at similar depth and thus reduce image softening due todefocus of one or more stages.

According to another exemplary embodiment of the present invention, FIG.5 shows a setup 500 with an imaging sensor 501 that comprises a firstpixel line 508 and a second pixel line 509 that are separated by thenon-photosensitive gap 506. First TDI block 502, second TDI block 503,third TDI block 505 and 128th TDI block 504 respectively comprise fourlines of pixels. Interruption 516 is shown for the pixel lines as pixellines are much longer than shown here in FIG. 5. As has been describedbefore, the pixel lines may consist of several thousand pixels, forexample, 4000 or more pixels.

FIG. 5 shows also shows two TDI engines 510, 513 which are positioned onthe imaging sensor and are thus part of the imaging sensor. Such a TDIengine is configured to carry out any of the known and herein mentionedTDI procedure. In this way, TDI is done on the chip. Also otherembodiments are comprised by the present invention, in which the TDIprocedure is carried out off the chip, for example by an externalcomputer. The eight input and output taps 511 and 514 are the standardpins for connecting the sensor to a databus. Optionally, the 24 inputand output taps 512, 515 can be used in case a higher bandwidth isdesired by the user.

FIG. 6 schematically shows a flow diagram of a method according to anexemplary embodiment of the present invention. In detail, a read-outmethod for generating seamless 2D or 3D images is presented in FIG. 6while changing capture depth during the scanning. The method is capableof simultaneously capturing image data at multiple depths of the sample.The method uses an imaging device like, for example, the devicedescribed in the context of FIG. 2. The used imaging device has anoptical axis and comprises an imaging sensor 300, 400 titled withrespect to the optical axis. Such an imaging sensor of the imagingdevice comprises a first pixel line 410 comprising a plurality of pixels415, 416 and at least a second pixel line 411 comprising a plurality ofpixels 418, 419. The first and second pixel lines have a differentoptical path length to the sample along the optical axis of the imagingdevice and are offset with respect to each other along the scandirection by an offset 402.

The method shown in FIG. 6 teaches to scan the sample along a scandirection X′ which is substantially perpendicular to a main extensiondirection Y of the first and second pixel lines of the imaging sensor,i.e., step S1. The second step S2 defines capturing a first image of thesample from the first pixel line, previous to step S3 which requiressimultaneously capturing a second image of the sample from the firstpixel line and capturing a third image of the sample from the secondpixel line. Moreover, in step S4 capturing images of the sample from thesecond pixel line is continued. Stopping capturing images of the samplefrom the first lines of pixels is done in step S5.

The method depicted in FIG. 6 with steps S1 to S5, therefore, can beseen as a read-out method which is capable of generating seamless 2D or3D images while changing capture depth during the scanning. This allowsfor a fast image acquisition of not perfectly flat and/or volumetricsamples. The calculation to determine how long it takes to bridge therespective offset, i.e., the gap between adjacent pixel lines duringscanning, can be done by the skilled person without a problem, and canbe implemented in the imaging device disclosed herein. Starting from thedistance between the two cameras, i.e., the two pixel lines, inprojection one knows to how many pixels the distance relates. With theexposure frequency, i.e., the line rate, one knows how many pixels arein the offset/gap.

The imaging device carrying out this method is capable of determiningwhen to start the simultaneous image capturing. In particular, in anexemplary embodiment of the present invention it is detected whether achange in acquisition depth is required, and a corresponding actuationof the second pixel line based on the detection that a change inacquisition depth is required id carried out by the imaging deviceautomatically. Many different technical means can be used for thedetection whether a change in acquisition depth is needed, as has beendescribed herein before.

The method explained with FIG. 6 allows for a fast image acquisition ofnot perfectly flat and/or volumetric samples, as shown in FIG. 1, andcan be specifically applied in digital slide scanners, e.g. for digitalpathology, but also in other technical fields. With this method of FIG.6 it is possible to capture a seamless image, which would not bepossible without the temporary dual, i.e., simultaneous readout, becausea change in line sensor results not only in a change of acquisitiondepth, but also in a translation along the scan direction X′. Thislatter translation causes either a gap in the image, or a repetition ofimage data. The method as described before is necessary to prevent thegap. For the repetition, part of the image data can be discarded, but nodual acquisition is needed.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from the study of the drawings, the disclosure, and theappended claims. For example, as explained previously the inventiondescribed in this application encompasses other configurations in whichthe sensor is not tilted with respect to the optical axis and in whichthe imaging system is arranged such that this sensor can image anoblique cross section of the sample. Thus, the generation of saiddifferent optical path lengths from the sample to the sensor may be madeusing other techniques well-known in the art such as inserting anoptical element, for instance a prism, in the light path.

In the claims the word “comprising” does not exclude other elements orsteps and the indefinite article “a” or “an” does not exclude aplurality. A single processor or other unit may fulfil the functions ofseveral items or steps recited in the claims. The mere fact that certainmeasures are recited in mutually different dependent claims does notindicate that a combination of these measures cannot be used toadvantage. Any reference signs in the claims should not be construed aslimiting the scope of the claims.

The invention claimed is:
 1. Method for simultaneous capture of imagedata at multiple depths of a sample, using an imaging device having anoptical axis, the device comprising an imaging sensor, the image devicebeing arranged such that the sensor can image an oblique cross-sectionof the sample, the imaging sensor having: a first pixel line comprisinga plurality of pixels, a second pixel line comprising a plurality ofpixels, the first and second pixel lines have a different optical pathlength to the sample along the optical axis of the imaging device, thefirst pixel line and the second pixel line are offset with respect toeach other along the scan direction by an offset, the method comprisingthe steps scanning the sample along a scan direction (X′) which issubstantially perpendicular to the optical axis and a main extensiondirection (Y) of the first and second pixel lines (S1), capturing afirst image of the sample from the first pixel line (S2), simultaneouslycapturing a second image of the sample from the first pixel line andcapturing a third image of the sample from the second pixel line (S3),continuing capturing images of the sample from the second pixel line(S4), stopping capturing images of the sample from the first pixel line(S5).
 2. Method according to claim 1, wherein the offset is either afirst non-photosensitive gap between the first and the second pixellines or a non-capturing pixel line between the first and the secondpixel lines which non-capturing pixel line is deactivated.
 3. Methodaccording to claim 2, wherein the non-photosensitive gap extendsparallel to the first and second pixel lines.
 4. Method according toclaim 1, wherein the steps S1 to S5 are carried out during scanning thesample.
 5. Method according to claim 1, wherein the simultaneouscapturing of the second image and the third image is carried out for aslong as it takes to bridge the offset between the first pixel line andthe second pixel line during scanning.
 6. Method according to claim 1,further comprising the steps detecting whether a change in acquisitiondepth is required, and actuating the second pixel line based on thedetection that a change in acquisition depth is required.
 7. Methodaccording to claim 6, wherein a method of predicting an optimal focusposition is used for detecting whether a change in acquisition depth isrequired.
 8. Method according to one of claim 6, the imaging sensorfurther comprising a third pixel line comprising a plurality of pixels,wherein the first, second and third pixel lines each have a differentoptical path length to the sample along the optical axis of the imagingdevice, wherein the first pixel line and the third pixel line are offsetwith respect to each other along the scan direction (X′) by an offset,wherein the first pixel line is located between the second and the thirdpixel line, the method further comprising the step actuating the secondpixel line in case it has been detected that an increase in acquisitiondepth is required or actuating the third pixel line in case it has beendetected that a decrease in acquisition depth is required.
 9. Methodaccording to claim 1, further comprising the step discarding repetitiondata acquired by the first pixel line and/or the second pixel line. 10.Method according to claim 9, further comprising the step generating afinal image of the sample based on the captured images after therepetition data was discarded.
 11. Method according to claim 10, whereinthe imaging system is an imaging sensor comprising a 2D array of pixelsin an orthogonal XY coordinate system, the 2D array of pixels of thesensor comprising the first, second and a third lines of pixels, andwherein each of the first, second and third pixel lines extends alongthe Y direction.
 12. Method according to claim 1 for generating a threedimensional image of the sample, the method further comprising the stepscapturing the first image of the sample from a first set of lines ofpixels comprising the first pixel line, simultaneously capturing thesecond image of the sample from the first set of lines of pixels andcapturing a third image of the sample from a second set of lines ofpixels comprising the second pixel line, continuing capturing images ofthe sample from the second set of lines of pixels, and stoppingcapturing images of the sample from the first set of lines of pixels.13. An imaging system for simultaneous capture of image data of a sampleat multiple depths, the system comprising a first pixel line comprisinga plurality of pixels, a second pixel line comprising a plurality ofpixels, wherein the device is configured to scan the sample along a scandirection (X′), wherein the first pixel line and the second pixel lineare offset with respect to each other along the scan direction by anoffset, wherein the imaging system is configured to capture a firstimage of the sample from the first pixel line, wherein the imagingsystem is configured to simultaneously capture a second image of thesample from the first pixel line and capturing a third image of thesample from the second pixel line, and wherein the imaging system isconfigured to continue capturing images of the sample from the secondpixel line and is configured for stopping capturing images of the samplefrom the first lines of pixels.
 14. An imaging system according to claim13, wherein the first and second pixel lines are part of an imagingsensor comprising a 2D array of pixels in an orthogonal XY coordinatesystem, wherein the first and second pixel lines extend along the Ydirection, the imaging sensor further comprising a firstnon-photosensitive gap between the first and the second pixel lines,wherein the first non-photosensitive gap extends along the Y direction,and wherein read out electronics of pixels of the first pixel lineand/or of pixels of the second pixel line are positioned in the firstnon-photosensitive gap.
 15. An imaging system according to claim 14,wherein at least one of the following components is positioned in thefirst non-photosensitive gap or in a further non-photosensitive gap ofthe sensor: current voltage converters of pixels of at least one of thefirst and the second pixel line, a logic of the imaging sensor, and aconnective circuitry of the imaging sensor.
 16. An imaging systemaccording to claim 15, wherein the imaging sensor is titled with respectto the optical axis.